Refining sentence similarity with discourse information in dialog system

The ability to accurately judge the similarity between sentences is important for dialog system development in various areas such as utterance verification, context reasoning, utterance clustering. However, standard text similarity measures fail when directly applied to dialog sentences which are usually very short and have many ungrammatical omissions and inversions. This paper presents a method for sentence similarity refining method using discourse similarity of dialog sentences. First, we propose a novel discourse similarity based on the dialog act taxonomy. Given discourse similarity, we then present a novel way of rescoring original sentence score by explicitly adding discourse score to it. Experiments on test data sets demonstrate that the proposed measure significantly outperforms traditional similarity scoring measures.